Multi-Phase Redistribution: A Communication-Efficient Approach to Array Redistributionz

Distributed-memory implementations of several scientific applications require array redistribution. Array redistribution is used in languages such as High Performance Fortran to dynamically change the distribution of arrays across processors. Performing array redistribution incurs two overheads an indexing overhead for determining the set of processors to communicate with and the array elements to be communicated, and a communication overhead for performing the necessary irregular all-to-many personalized communication. In this paper efficient runtime methods for performing array redistribution are presented. To reduce the indexing overhead, precise closed forms for enumerating the processors to communicate with and the array elements to be communicated are developed for two special cases of array redistribution involving blockcyclically distributed arrays. The general array redistribution problem for block-cyclically distributed arrays can be expressed in terms of these special cases. Using the developed closed forms, a distributed algorithm for scheduling the irregular communication for redistribution is developed. The generated schedule eliminates node contention and incurs the least communication overhead. The scheduling algorithm has an asymptotically lower scheduling overhead than techniques presented in the literature. Based on the developed closed forms, a cost model for estimating the communication and the indexing overhead for array redistribution is developed. Using this model, a multi-phase approach for reducing the communication cost for array redistribution is presented. The key idea is to perform the redistribution as a sequence of redistributions such that the total cost of the sequence is less than that of direct redistribution. Algorithms for determining the sequence of intermediate data distributions which minimizes the total redistribution time are developed. Extensions of the developed closed forms and algorithms to perform array redistribution of multi-dimensional arrays are presented. Experimental results on the Cray T3D and IBM-SP2 demonstrate the validity of the developed cost models and the efficacy of the multi-phase approach for array redistribution.